\n",
"\n",
"## Learn natural language processing. From A to Z. From zero to hero. Fast.\n",
"\n",
"Copyright, NLP from scratch, 2023.\n",
"\n",
"[nlpfromscratch.com](https://www.nlpfromscratch.com)\n",
"\n",
"------------"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gmAd3hBe94HF"
},
"source": [
"# Webinar #2 - What the Heck is an LLM?"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "uuznFjxWVV0n"
},
"source": [
"## Introduction 🎬\n",
"In this notebook, we will see several different applications of Large Language Models (LLMs), and show how they can be leveraged the open source libraries from [Hugging Face](https://huggingface.co/).\n",
"\n",
"This notebook is best run in [Google Colab](https://colab.research.google.com/), where the majority of dependencies are already installed. However, if you wish to run the notebook locally, please follow the [directions for setting up a local environment](https://drive.google.com/file/d/1EV1seK-dUHRCzj2EDuu3ETAhUyjzOGRd/view?usp=drive_link) and you may then download the notebook as a `.ipynb` and run in either Jupyter or Jupyterlab.\n",
"\n",
"Though Google Colab comes with many useful data science libraries included by default (including Pytorch), the Hugging Face libraries are not, so we will first install those here using `pip`, as they will be used in the remainder of the notebook.\n",
"\n",
"- The `transformers` library, for general usage of transformer models\n",
"- The `datasets` library, for working with datasets hosted on Hugging Face\n",
"- The `diffusers` library, for working with diffusion models for image generation\n",
"- The `accelerate` library, for using GPU for inference"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "fjcuFD9_-ajD",
"outputId": "2b36db74-4093-48f7-ce16-95adceb2b372"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: transformers in /usr/local/lib/python3.10/dist-packages (4.38.2)\n",
"Collecting datasets\n",
" Downloading datasets-2.18.0-py3-none-any.whl (510 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m510.5/510.5 kB\u001b[0m \u001b[31m3.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting diffusers\n",
" Downloading diffusers-0.27.2-py3-none-any.whl (2.0 MB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.0/2.0 MB\u001b[0m \u001b[31m19.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting accelerate\n",
" Downloading accelerate-0.29.3-py3-none-any.whl (297 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m297.6/297.6 kB\u001b[0m \u001b[31m14.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers) (3.13.4)\n",
"Requirement already satisfied: huggingface-hub<1.0,>=0.19.3 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.20.3)\n",
"Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (1.25.2)\n",
"Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers) (24.0)\n",
"Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (6.0.1)\n",
"Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (2023.12.25)\n",
"Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers) (2.31.0)\n",
"Requirement already satisfied: tokenizers<0.19,>=0.14 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.15.2)\n",
"Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.4.2)\n",
"Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers) (4.66.2)\n",
"Requirement already satisfied: pyarrow>=12.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (14.0.2)\n",
"Requirement already satisfied: pyarrow-hotfix in /usr/local/lib/python3.10/dist-packages (from datasets) (0.6)\n",
"Collecting dill<0.3.9,>=0.3.0 (from datasets)\n",
" Downloading dill-0.3.8-py3-none-any.whl (116 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m116.3/116.3 kB\u001b[0m \u001b[31m12.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets) (2.0.3)\n",
"Collecting xxhash (from datasets)\n",
" Downloading xxhash-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m194.1/194.1 kB\u001b[0m \u001b[31m3.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting multiprocess (from datasets)\n",
" Downloading multiprocess-0.70.16-py310-none-any.whl (134 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m7.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: fsspec[http]<=2024.2.0,>=2023.1.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (2023.6.0)\n",
"Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets) (3.9.3)\n",
"Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.10/dist-packages (from diffusers) (7.1.0)\n",
"Requirement already satisfied: Pillow in /usr/local/lib/python3.10/dist-packages (from diffusers) (9.4.0)\n",
"Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate) (5.9.5)\n",
"Requirement already satisfied: torch>=1.10.0 in /usr/local/lib/python3.10/dist-packages (from accelerate) (2.2.1+cu121)\n",
"Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.3.1)\n",
"Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (23.2.0)\n",
"Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.4.1)\n",
"Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (6.0.5)\n",
"Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.9.4)\n",
"Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (4.0.3)\n",
"Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.19.3->transformers) (4.11.0)\n",
"Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.3.2)\n",
"Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.6)\n",
"Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2.0.7)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2024.2.2)\n",
"Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (1.12)\n",
"Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (3.3)\n",
"Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (3.1.3)\n",
"Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch>=1.10.0->accelerate)\n",
" Using cached nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB)\n",
"Collecting nvidia-cuda-runtime-cu12==12.1.105 (from torch>=1.10.0->accelerate)\n",
" Using cached nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB)\n",
"Collecting nvidia-cuda-cupti-cu12==12.1.105 (from torch>=1.10.0->accelerate)\n",
" Using cached nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (14.1 MB)\n",
"Collecting nvidia-cudnn-cu12==8.9.2.26 (from torch>=1.10.0->accelerate)\n",
" Using cached nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl (731.7 MB)\n",
"Collecting nvidia-cublas-cu12==12.1.3.1 (from torch>=1.10.0->accelerate)\n",
" Using cached nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl (410.6 MB)\n",
"Collecting nvidia-cufft-cu12==11.0.2.54 (from torch>=1.10.0->accelerate)\n",
" Using cached nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl (121.6 MB)\n",
"Collecting nvidia-curand-cu12==10.3.2.106 (from torch>=1.10.0->accelerate)\n",
" Using cached nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl (56.5 MB)\n",
"Collecting nvidia-cusolver-cu12==11.4.5.107 (from torch>=1.10.0->accelerate)\n",
" Using cached nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl (124.2 MB)\n",
"Collecting nvidia-cusparse-cu12==12.1.0.106 (from torch>=1.10.0->accelerate)\n",
" Using cached nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl (196.0 MB)\n",
"Collecting nvidia-nccl-cu12==2.19.3 (from torch>=1.10.0->accelerate)\n",
" Using cached nvidia_nccl_cu12-2.19.3-py3-none-manylinux1_x86_64.whl (166.0 MB)\n",
"Collecting nvidia-nvtx-cu12==12.1.105 (from torch>=1.10.0->accelerate)\n",
" Using cached nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (99 kB)\n",
"Requirement already satisfied: triton==2.2.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (2.2.0)\n",
"Collecting nvidia-nvjitlink-cu12 (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.10.0->accelerate)\n",
" Using cached nvidia_nvjitlink_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl (21.1 MB)\n",
"Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.10/dist-packages (from importlib-metadata->diffusers) (3.18.1)\n",
"Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2.8.2)\n",
"Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2023.4)\n",
"Requirement already satisfied: tzdata>=2022.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2024.1)\n",
"Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.2->pandas->datasets) (1.16.0)\n",
"Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.10.0->accelerate) (2.1.5)\n",
"Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.10.0->accelerate) (1.3.0)\n",
"Installing collected packages: xxhash, nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, dill, nvidia-cusparse-cu12, nvidia-cudnn-cu12, multiprocess, nvidia-cusolver-cu12, diffusers, datasets, accelerate\n",
"Successfully installed accelerate-0.29.3 datasets-2.18.0 diffusers-0.27.2 dill-0.3.8 multiprocess-0.70.16 nvidia-cublas-cu12-12.1.3.1 nvidia-cuda-cupti-cu12-12.1.105 nvidia-cuda-nvrtc-cu12-12.1.105 nvidia-cuda-runtime-cu12-12.1.105 nvidia-cudnn-cu12-8.9.2.26 nvidia-cufft-cu12-11.0.2.54 nvidia-curand-cu12-10.3.2.106 nvidia-cusolver-cu12-11.4.5.107 nvidia-cusparse-cu12-12.1.0.106 nvidia-nccl-cu12-2.19.3 nvidia-nvjitlink-cu12-12.4.127 nvidia-nvtx-cu12-12.1.105 xxhash-3.4.1\n"
]
}
],
"source": [
"!pip install transformers datasets diffusers accelerate"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7mElaFxm94HH"
},
"source": [
"## Generating Text: Working with GPT ✍"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qeqMEGd394HM"
},
"source": [
"One of the most well-known transformer models for text generation is that of the Generative Pre-trained Transformer ([GPT](https://en.wikipedia.org/wiki/Generative_pre-trained_transformer)) family of models, made famous by [OpenAI](https://openai.com/).\n",
"\n",
"The first GPT model was released in 2018 and subsequent models, GPT-2 and GPT-3 released in 2019 and 2020, each larger in the number of parameters from the proceeding by a factor of ~10x. GPT-3 was the precursor to InstructGPT, from which OpenAI created the immensely popular and disruptive ChatGPT. GPT-4 was released in March of 2023 and has around 1.8T parameters and represents the current state-of-the-art in LLMs.\n",
"\n",
"The model weights for GPT-3 onward have not been released and are proprietary, but those for GPT-2 are publicly available. Due to computational constraints (even with GPU), here we will work with a much smaller GPT-2 model of only ~124M parameters.\n",
"\n",
"With only a few lines of code, we can download the model from the Hugging Face platform, instantiate it in a [pipeline](https://huggingface.co/docs/transformers/main_classes/pipelines), and then generate text based on an input string:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 417,
"referenced_widgets": [
"65b39a2145fd4dafa0ad6f564ea584c0",
"9ebf3e2b08b848e3a17a53044c105d0c",
"febb3d793a2642b99e4938bf95447845",
"f4083673a303494fb0dd2f3b8071c330",
"dcea70d3308d48b3b0eaf82c38524cc7",
"2fb9d44cb7b54c46aeb04892077cb3ba",
"f5477dc1263948639829a09f4f85aace",
"de0bdccc0d684e698055bd89981a7f6c",
"6ae62bd07621470e91464a3755dd8268",
"db54b1f0a1804c818b3e5eae80b37545",
"c5d56aa042d4484e8ec4af3a64273be4",
"329b4bba9c0e41579541deebd39cb902",
"0376e2b022314dcebe853bf3321b10f1",
"0f5da300bb334b798ee158d70fec982d",
"a26714dd61b34f8ca2379f788f157159",
"e17d94a485bb4adda4500c9dc78e4c64",
"29ed28fe90be46a5a6526bb7bdccc96d",
"d0aff26389c54082ac73b54109a90835",
"05d19a6582e9437cb17d80a6ec0cc742",
"f0dea3528e3e4fc6b203df2587f0cc04",
"ebe65408d2554adba60c9542952e07f7",
"ce1f4124bd0a44f798dfc2c31d09d4e2",
"1e6b23c17e184adebeb0dbbc1f6b1454",
"c4ee1afb25b140fd868dce85044eb4d2",
"485427a68f1f48b184bbd193242a649f",
"ea7338464b1a41609ebdd2d331c3de65",
"e19ebf3d9d5141fab41987eb7caf9fe5",
"61b81ecf40b44d6c81883cada7e0be44",
"55557865c21544deaed5480e50125855",
"f824b26385d9459993ae58edda56fa47",
"4f9188e183be4f4ab37ef094b0d82a3e",
"23865a24ead44c08960410bd3de76cf0",
"0e57c05c7c034331b695bcacf490af58",
"5dbbab062f12464696e8843fc459c53c",
"75f2a66d57b347e4b9a4c524b9a93f69",
"96091b8d542e410ea2570cafebb079fe",
"fb35e5947b4e4e08a2698d94f41fc936",
"c79e146f01d3429f8fdc8850590b8c95",
"048faccdae4044c681ec42aae3bf3786",
"b2890b6644a34c75a104518f7c5b71ce",
"7062cb78c7d54e32aee2da0d86b9a49b",
"dc569ecfdc494e3c94b8721304c7fd8d",
"ede22fba48144f1f82b3084c0c820723",
"9df363fb99c44e73904267153cd2ee77",
"39ef842ea90541ebb85643160b1f1276",
"60d9c9dfae8849d7a2083d5aa165c3dd",
"1683d1fa2ad647cf86f9ae88bcddd3ef",
"b318f872f92541b2a7941b2cdf13641f",
"dad2278bbb404d1195515234d552290d",
"0171e4d1efb34ce0920cb12da7d9a2bf",
"3bbbd3b0499f4bc2856f323092c7dac0",
"c0173dc36e8d47ca864d0dcec27d6f7e",
"927f0251c32241f99d5070795754f299",
"802cb3af3f454671945d39dd1a32de1c",
"b63e79aa356048439a61da1813fc86cc",
"1f6d28d670994895bfb30f3e9eb4cadb",
"a70f06a6645f4145b299aef3996dddd2",
"da679a0b605c4573b799e388a3da7e64",
"4752631dec0f421084d987f431227875",
"08491b7037c24a3dbf65eff844cb346e",
"b6f302a383bf412bb8c93c7f622bf812",
"2eebe8e2ca264d238720c05dd2a1c927",
"4965060fa163491ca9b73d758c0a5634",
"a5740b5cd8ce4681ab0a92580c0a0ef9",
"d29825df8102462fa293f1500295d5a3",
"fff850f4e0fb4d3c8f670be4ef028719",
"30d0e226774848f08135fa2f1f304963",
"d40545c27c8d40229109631842e71c54",
"9ee80398008a4ef6b87c0edc100843a9",
"e8be9b3bcdca4408adf2a791fb7cc3cd",
"72786c2d558f48c48002d663b7fa7933",
"6532f85c4a294db59571aac2ec3c054b",
"1c9a627f846e430e8f3991d9381ba4eb",
"52c04e6a53f748b2844d5e360d8fb8ee",
"82a53461f0f54efd9cd4830b55c08bcc",
"982698bba37c4612bc4b96394c334af3",
"57a3fa45774d471fa04df3b11123dd92"
]
},
"id": "QY-tMQSE94HP",
"outputId": "80eaec76-0b1d-4bb6-8e4c-215ac673ca6b"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:88: UserWarning: \n",
"The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
"To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
"You will be able to reuse this secret in all of your notebooks.\n",
"Please note that authentication is recommended but still optional to access public models or datasets.\n",
" warnings.warn(\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "65b39a2145fd4dafa0ad6f564ea584c0",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"config.json: 0%| | 0.00/665 [00:00"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/markdown": [
"The rain in Spain falls mainly in the plain, and furthermore with their small numbers and high rate of development they have little ability to provide for themselves.\n",
"\n",
"There's only one major urban area with"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/markdown": [
"The rain in Spain falls mainly in the plain, and furthermore it can break down the snow that is on top: this is the \"sunny place\", and we see that the rain gets more heavy"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Display the outputs nicely\n",
"from IPython.display import Markdown\n",
"\n",
"for output in outputs:\n",
" display(Markdown(output['generated_text']))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "74n00xq-94HQ"
},
"source": [
"## Stable Diffusion and Generating Synthetic Images 🎨"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9HB7qO79_d0y"
},
"source": [
"We can also programmatically generate synthetic media using multimodal NLP and the power of Large Language Models (LLMs). [Stable Diffusion](https://en.wikipedia.org/wiki/Stable_Diffusion) is a very popular series of text-to-image models for this purpose which was released publicly by [Stability AI](https://stability.ai/) in August of 2022. Stable Diffusion is not a transformer architecture (and so some would say is not technically an LLM), though there are other cutting-edge image generation models based on the transformer architecture such as OpenAI's [DALL-E](https://openai.com/dall-e-2).\n",
"\n",
"Stable Diffusion can be used through managed services such as [Clipdrop](https://clipdrop.co/stable-diffusion), which was used to generate the image below:"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pBN1Bfao94HR"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "z7qx6nR3u-hP"
},
"source": [
"It is also available through a Hugging Face space here: https://huggingface.co/spaces/google/sdxl"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "XIIhFNzt94HR"
},
"source": [
"Alternatively, if we are more interested in programmatically generating content as part of an application, we can use stable diffusion as part of the Hugging Face [diffusers](https://huggingface.co/docs/diffusers) library, as the model is publicly available. Here, we will use the original [Stable Diffusion 1.4 model](https://huggingface.co/CompVis/stable-diffusion-v1-4):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 81,
"referenced_widgets": [
"90ed60a2459a464eaef5f1627103928f",
"5f31a6571ee54b1da580983b4c1930f5",
"e1aaa48ed81f4e8a993d6666c20c1c17",
"e6049cea248249a3bfebc9beb5e08ec2",
"4c3f2623f6ff436cb627b51ac136876a",
"aacf898fc48c4d738b143891be48bfe4",
"d63a0e96ef3445fdbd7cd336ee69dcef",
"19b41b3571a34ab585e1acff6662482c",
"c5fbe80116f34f42b5fdb0a9ec8e8b62",
"e2f0d8df3ad2450b96067ddddf34e579",
"edccb56e75514824a1179210d3be3281",
"c962ef35c42441a690cdd829a4366110",
"17f3b9a3a71a44f0a917b725b3eb50b5",
"625c66903ed64208abaa5bf3b647d59f",
"d910912dfc2a4920b665e80c2fb071c7",
"bc51271952dc4412872189eef098fc60",
"4c53d9e6f0fe4e2c9c80eade9450120f",
"17b3a3de20bd4212be694bb552f157de",
"1e4d434abde642569c2449f27249a014",
"305379eeb3c9403da00aa33d97feaf84",
"b025af452faf4f0eab00c6e901592e15",
"3761cf2642cf4ee7b06b49cd50ff206b"
]
},
"id": "FFaFqBDSJqK0",
"outputId": "ad0fda05-89fc-4074-b253-d2d7dc85fe34"
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "90ed60a2459a464eaef5f1627103928f",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Loading pipeline components...: 0%| | 0/7 [00:00"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import cv2\n",
"from google.colab.patches import cv2_imshow\n",
"\n",
"img = cv2.imread('image.png')\n",
"\n",
"cv2_imshow(img)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kvYxiZ1R94HW"
},
"source": [
"### Whisper and Automatic Speech Recognition (ASR) 🗣"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3AgPwg2D94HW"
},
"source": [
"Another area of multimodal NLP is that of transcribing audio. This is known historically as text-to-speech, but in the domain of machine learning is usually referred to as an automatic speech recognition (ASR) task.\n",
"\n",
"OpenAI has developed its proprietary [Whisper model](https://openai.com/research/whisper), which, while making available to call through APIs, has also released the model weights and code publicly, and so can also be freely used through Hugging Face by downloading the model and using the associated [wrapper classes](https://huggingface.co/docs/transformers/model_doc/whisper).\n",
"\n",
"The Whisper model is quite complex, combining many different speech recognition tasks ( multilingual speech recognition, speech translation, spoken language identification, and voice activity detection) into a single model, however in simple terms it takes audio, transforms this into image data of spectrograms, and then uses these to predict the output tokens of the transcription."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "jCz-xtrw94HW"
},
"source": [
"Here, we will need some audio data to work with, so we will use that which Hugging Face provides in the \"Hello World\" example for Whisper, samples of audio from [Librispeech ASR corpus](http://www.openslr.org/12).\n",
"\n",
"First, we will import `datasets` and load the sample dataset of audio from Hugging Face:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 202,
"referenced_widgets": [
"03c7d1454926416cb778dc30903d38c9",
"a8914ad863f6429d90a0a043b64a5870",
"21ce237066df42e1a1eca5e71e7af003",
"781148411d5f4cf3ba8a5123923dcc38",
"e824f2ea3ce240dd84dd65253730f388",
"ef99a9424daf4c0e82a79e9800747cf4",
"d00b6cfae82a49d0b07dc33e81fa5a0e",
"71b8eeaab91b44f991bf571ce00e646f",
"4bb362e003f5414c9de169d5c4951033",
"c1389086de334e9eade7d838d21bc2cb",
"48474ca3a7a44312ade7428feef939f8",
"ef3d468c9aa349d2993afeaee79c6df8",
"52a9a40c4d8945e58d2ac6cb0f889d92",
"1360be0c6f094c1d82cef0ed9fbae841",
"6f3fcbde60c54fec85a1a782eae536a6",
"75d0a6e0eda94f9296d0065690e8f486",
"de6dd41e75734ec3933e0f99d94af0cf",
"841ffba4d53244bb8515f4b5f0b436ea",
"32b51262a3f14a0881c13938416144c5",
"7c6399edf4ed4af38de9be79aaf2569d",
"8db16f5dc49d467fa32c768022e8f2fb",
"f4cfb348c9ac494faf287bd70a1e3d4a",
"c783a749736f48d69fa93a4e91763e8b",
"b2ccf6837b9241d9a41f2e7debd78a61",
"ec8fc0c47c98409abc6903952407ffe6",
"2aefba89dbf5453da6479a71300bb7f0",
"26a59cac6cf54bd59a78005de721e4f7",
"0fdcff7983ae4147bf7c0706945e1ea1",
"bc7786fc69e342f6b57a49701290fb76",
"ff8d9a3d5e7b4787ba359fe596d03c82",
"0c92cd86e7cc4a4e9dd6fe7548335a31",
"4ca4552d4f83406c8fb7430a34883938",
"f04fa644f79742088ea096703a8f9193"
]
},
"id": "qUBy3zkT94HW",
"outputId": "d3cf950b-dbb5-412a-a2bd-4868ceaf6f2d"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.10/dist-packages/datasets/load.py:1461: FutureWarning: The repository for hf-internal-testing/librispeech_asr_dummy contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/hf-internal-testing/librispeech_asr_dummy\n",
"You can avoid this message in future by passing the argument `trust_remote_code=True`.\n",
"Passing `trust_remote_code=True` will be mandatory to load this dataset from the next major release of `datasets`.\n",
" warnings.warn(\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "03c7d1454926416cb778dc30903d38c9",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Downloading builder script: 0%| | 0.00/5.17k [00:00\n",
" \n",
" Your browser does not support the audio element.\n",
" \n",
" "
],
"text/plain": [
""
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from IPython.display import Audio\n",
"Audio(sample['array'], rate=sample['sampling_rate'])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VNVwMfhe94HY"
},
"source": [
"Now we can pass the audio into the processor and model, and it will transcribe the audio. The Whisper model comes in a variety of sizes, here we are usin [whisper-small](https://huggingface.co/openai/whisper-small) which has 244M parameters, compared with largest version which has ~1.5B."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 424,
"referenced_widgets": [
"16fb82bd3f754085993d01ff87a1d595",
"f1f08036a41b4cdaa0ddad39ef38df96",
"ab32357fa6d248c0a0fe0cadd4dec548",
"f3203d0f7b75444ca5d1aa88cce756f3",
"1024aefa5d8d425da480ec18e5467ff1",
"a711647aa5014ddca3c77f1b5e54d3ee",
"bca0b71cfcae40969d1a798b76baaa45",
"a894779d782647ab881026a59f667b41",
"fce3195a82de41fdb4ac4c595d49f9f1",
"17f60176134e46d194cf5249ca8bb27b",
"c29173b8422440d3a1e703d250401083",
"cb2bfd1060c445d5ad92afb192ab2440",
"f9213ee5f6544209a7787b3c3335e783",
"5189ef7b7cbe486db62443825837f25d",
"f7833d00c020403da8f6b513f6d076b4",
"21192f35d5194645950a9d8c98865d62",
"98417a6535c6446794c7f3cd129b74b9",
"5c4bafcaed2241748a8c459e34254a3f",
"5ba42b7254ab425aa4a4f3ad22d297c4",
"86fbd4bb5e9745338ee883caee7013d0",
"8d9ad822cad24419ba83969091fa868f",
"32745be795ef4d39b7872cee463a3721",
"f84798c2a0db43bf92933e0c287edb1c",
"e763fefb606d4b63ad87408ac304ce2b",
"4b7e487b1ad7443e8bb34b97a74e3f00",
"b16b5ff1175a48dd8d0f40337f53ac3d",
"547ce622b14742a2b2c3d860d4925397",
"58836af6b97c43858c7fd321c55cf283",
"95ef3d2b58f048138d686f1d37b83fd6",
"e596ca68ea81439985be939a83d1c459",
"b551eea244af4505b20f408e5c5db592",
"28ffe02a9e53487c97f790ee26703768",
"57f5e96a70264a858cd9e3b156c54743",
"edd5578773dd452a9c97f0278567fbf0",
"32ba74acf13d4694acadd3465d659eae",
"50ce2ea956f047a789bab0dd9c2d1233",
"02af60a130164029a02dd2d6a0bf6f14",
"464bed7afe694192bc22c7980bed2c53",
"78c3e64884214f70aa7541a81e699ce6",
"8a765c592ddf454ea38442bf90f9aa16",
"07dbd22e84574cf39fc6b812d9f28d1e",
"b886eb6bac9648d4be35a61ce627cec4",
"c21ec34298c9487db11178309a161c39",
"f369421cf93d4fc0b3140f4ec8b26f2b",
"28f7bcda8dee4ba8a06357ecb17d9280",
"f99b111f976a45e2be307b08b57ee41b",
"8b18c3c7bff9493a99b4f2d2302ef245",
"df000606e43847f1963a4ed446441571",
"5efb66879d114fb39ab249b74510e8ef",
"d6493c8ed9c648f48c62b4340ed1b272",
"451a7f9e58a74f899625500a1e6215f1",
"b5163a6743ea44f2b7a80185a5b2f30e",
"6d3981faf8bc4dab859799656e42cfb0",
"0c40915c90ba488f98d34d7e57c7654a",
"65dafd4ab58d4032b53b5564e0d401aa",
"3a97caffc37b422a8c1583f3be03a1b0",
"a1d47c8d064f41fd8c5cc8abf4cbdbed",
"ff652c55d7bd4828915496f5e891b13a",
"2dd18ad781ba4136a640c9e94c4fabaa",
"0e1d655a2f6e412e91989e140734095f",
"a6cff1bc0eb64a4e9496e3592c38e777",
"210686872fa846e2939cf96ca6b632ef",
"3e598616a0124e34b8bae7405de61317",
"40e341eba58148f387f510d6957465e1",
"daa3dc7b28f5439883513c90db87deaa",
"24a4e8a86f864fe18f677aa46c0c0666",
"3d647bcd3c2a4be092ce96866e70b922",
"4519fbde8811471488e4859f6b6f3d87",
"a6444cc0e44b42bba86c5ecf29970580",
"5cd44ac2106447128c0e12e47d2072dc",
"f3f0cdbaa414450c9eec856793a2b2b8",
"9e30b4f3938a4164a4cab59eef71d9cc",
"f8b26433c5064e6a88d222189d4cab77",
"44e189a2dbf449449340bdff5f0b1848",
"e245c865d10e484c8aedd23b786012a0",
"61c0e444c48b46238e86bd116fbee07e",
"8a9be9fa7eef44078133ab58d1ef2b97",
"ebc121d017b24180babbffab97f4a070",
"4b4f5b1c9acb43b38bf377e4575cf19c",
"a155eb5cea72404a8185e018f9e86c19",
"22c33443dcae4ddbb6d933105b8b4543",
"60d708ee1a3041b5b506d43471abe958",
"edcbc8f01a9544499efb6e5980885d63",
"47d5217300ee4f0f898c1014ee1d24f4",
"a687aab779cc40b9882ce28d34529051",
"80de7fc0957d493f93c626c3ad30bfd8",
"5769c09ddc424659ac609fbc83d7046b",
"4eed7f13773547918dcc29747d8c753d",
"bcd05a216ddb43738fdbf69da5a59d34",
"5f5ab26c834640dc81d96d16522a6ae1",
"c610f84f78504c1abeedf0b8fdb70bb2",
"2beeaa4f96624968a3034bd576107155",
"c8cdea3e940643f8a19b33f43ce0211e",
"4cbfbacac5ad4320824251f66059078e",
"ac6bfce5c56f4f3895e046aac77f5ef7",
"b85f5d54c14a4e9b8bd4ed552771a5c7",
"ed44ddf84b6742a5943b62871e44f4b4",
"6b067848f17e40ed8568ce4c72e98386",
"b4d9bd6a6cae4c498c8d215339fe922f",
"01f8be4037d8414bac2a4fcf0399aaed",
"17f54086937f473b9d48e01079efe0d4",
"6c23239789f04348a3a45e5a38eb3f97",
"9167684a1ef043f7ac2571e78262d8f5",
"6c3c6970be4e4b88b5ecd50c3c7793c7",
"6f94abacc5ca41c9a72385a6f14bb05a",
"ad43c53f52fe4a778ac2d7e0cb424e17",
"cb0c3b7351824bb982184e5660acc0cf",
"cc1547d87132463baf5bf127dbd936c4",
"3cb188d5b9fc45b8b833c60552c58c75",
"cd383f2fe4254b8e9b190e89156ae193",
"5a9c3781bc404848bfffe343bb1a0d9e",
"727a59f74ed54714acf13b77c723acea",
"686ea521c66c4ea492b216aa6b726ccd",
"cb4e6428b7d44aca9f2de683bb47e31d",
"c86b1c66de3341e49fc30eba71aead48",
"9da5cf061e3a4999af4845ea23ec1c41",
"afe34ac348bf4da58bba3888511f42c7",
"b1c7f0d880a74375bd6cf57d4a148046",
"249059eb3c8548a4a2cdb4b1b3054583",
"dc3d90a967bf42cdb71e16120aff4259",
"fe6ffe6cbd3142909edbbbe2c1d60667"
]
},
"id": "L8PyHI0d94HY",
"outputId": "4f32561c-7e1f-4323-8f83-76d1e5c3e89e"
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "16fb82bd3f754085993d01ff87a1d595",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"preprocessor_config.json: 0%| | 0.00/185k [00:00"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"display(Markdown(transcription[0]))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rD73ypeI0NXP"
},
"source": [
"We can see the transcription model is not perfect, if we compare with the original text:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
},
"id": "uj0Hpjcj0Sm0",
"outputId": "de3dbe5a-be14-4cd5-c0cc-650480f46ecb"
},
"outputs": [
{
"data": {
"application/vnd.google.colaboratory.intrinsic+json": {
"type": "string"
},
"text/plain": [
"'THEN THE POWERFUL TWIST THAT THRUST IT ASIDE IN AND UNDER THE GUARD'"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ds[72]['text']"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "cZMWwXyC0izd"
},
"source": [
"As such we would expect better performance from one of the larger model versions. Nonetheless, the model performs well and can quickly transcribe any spoken audio file in [many different languages](https://github.com/openai/whisper#available-models-and-languages)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "yC4JS7C094HY"
},
"source": [
"## Conclusion 🏁\n",
"\n",
"That concludes the webinar for today! I hope you've learned a lot about the large language model landscape and what is possible with the tools available to us today, and only a few lines of code here or there.\n",
"\n",
"I encourage you to further explore in the list of references and resources below, and to check out further opportunities for learning at https://www.nlpfromscratch.com/training.\n",
"\n",
"Thanks for coming!"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_k9JNl9d94HY"
},
"source": [
"### References and Further Reading\n",
"\n",
"- *Generative Pre-trained Transformer.* (n.d.). Wikipedia. Retrieved September 22nd, 2023. https://en.wikipedia.org/wiki/Generative_pre-trained_transformer\n",
"\n",
"- Ali, Fawad. (2023, April 11). *GPT-1 to GPT-4: Each of OpenAI's GPT Models Explained and Compared.* MakeUseOf.\n",
"https://www.makeuseof.com/gpt-models-explained-and-compared/\n",
"\n",
"- *Stable Diffusion.* (n.d.). Wikipedia. Retrieved September 22nd, 2023. https://en.wikipedia.org/wiki/Stable_Diffusion\n",
"\n",
"- OpenAI. (2022, September 21). *Introducting Whisper.* https://openai.com/research/whisper"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KsxOj0mxWXUw"
},
"source": [
"----\n",
"\n",
"
![](\"assets/banner.png\")